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p ! . Abstract 

' Given an ergodic finite-state Markov chain, let Mi w denote the 

mean time from i to equilibrium, meaning the expected time, starting 
from i, to arrive at a state selected randomly according to the equilib- 
rium measure w of the chain. John Kemeny observed that Mi w does 
not depend on starting the point i. The common value K = Mi w 
is the Kemeny constant or seek time of the chain. K is a spectral 
\ invariant, to wit, the trace of the resolvent matrix. We review basic 

■ facts about the seek time, and connect it to the bus paradox and the 

Central Limit Theorem for ergodic Markov chains. 



For J. Laurie Snell 



The seek time 



5_i ■ We begin by reviewing basic facts and establishing notation for Markov 

chains. For background, see Kemeny and Snell [3] or Grinstead and Snell 
[3], bearing in mind that the notation here is somewhat different. 

Let P be the transition matrix of an ergodic finite-state Markov chain. 
We write the entries of P using tensor notation, with being the prob- 
ability that from state i we move to state j. (There is some possibility 
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here of confusing superscripted indices with exponents, but in practice it 
should be clear from context which is meant.) The sequence of matrix pow- 
ers I, P, P 2 , P 3 , . . . has a (Cesaro) limit which we will denote by P°°. We 
have PP°° = P°°P = P°°. The rows of P°° are identical: 

(P°°)/ = ™ j - 

Like the rows of P, the row vector w is a probability distribution: > 0, 
Yjj w 3 = 1- w is the equilibrium distribution of the chain. The entry vfl tells 
the steady-state probability that the chain is in state j. 

The row vector w is, up to multiplication by a constant, the unique row 
vector fixed by P: 

X ' »■'■ 

i 

That makes it a row eigenvector corresponding to the eigenvalue 1. The 
corresponding column eigenvector is the constant vector: 

j 

Let Mij be the expected time to get from state i to state j, where we take 
Mjj = 0. The mean time from i to equilibrium is 

M iw = Y,M ij w j . 

j 

This tells the expected time to get from state i to a state j selected randomly 
according to the equilibrium measure w. 

John Kemeny (see jH 4.4.10], [3], p. 469]) observed: 

Theorem 1 Mi W doesn't depend on i. 

The common value of the M^'s, denoted K, is the Kemeny constant or 
seek time of the chain. 

Proof. Observe that the function M iw is discrete harmonic, meaning 
that it has the averaging property 

Y J P l ] M 3W = M lw . 

3 

The reason is that taking a step away from i brings you one step closer to 
your destination, except when your destination is i and the step begins a 
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wasted journey from i back to i: This happens with probability w % , and 
the expected duration of the wasted journey is i, because the mean time 
between visits to i is the reciprocal of the equilibrium probability of being 
there. Thus 

w i 

so 

But now by the familiar maximum principle, any function fi satisfying 

j 

must be constant: Choose i to maximize fi, and observe that the maximum 
must be attained also for any j where P^ > 0; push the max around until it 
is attained everywhere. So M iw doesn't depend on i. ■ 

Note. The application of the maximum principle we've made here shows 
that the only column eigenvectors having eigenvalue 1 for the matrix P are 
the constant vectors — a fact that was stated not quite explicitly above. 

The foregoing argument shows the mean time from i to equilibrium is 
constant — but what is its value? For this we return to Kemeny's original 
proof of constancy for Mi W , which involved writing an explicit formula for 
M iw , and noticing that it doesn't depend on i. 

Define the resolvent or fundamental matrix or Green's function 

Z = (I- P°°) + (P- P°°) + (P 2 - P°°) + ... 
= (/ - (P - P 00 ))- 1 - P°° . 

Please be aware that this resolvent Z differs from the variant used by Kemeny 
and Snell [1] and Grinstead and Snell [3], which with our notation would be 
(I — (P — P 00 ))^ 1 . As others have observed (cf. Meyer [B]; Aldous and 
Fill pQ), for the version of Z we use here, the entries ZJ have the natural 
probabilistic interpretation as the 'expected excess visits to j, starting from 
i, as compared with a chain started in equilibrium'. Accordingly we have 

i 

and 

E z/ = o. 

3 
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Since Z? measures excess visits to j starting at i, relative to starting in 
equilibrium, we obviously have Zj J > Z/ , because to make excess visits to 
j starting from i you first have to get to j. And the discrepancy Z- 3 — Z? 
is just Mijiv 3 , because in equilibrium this is the expected number of visits 
to j over an interval of expected length My. From this we get the familiar 
formula 

J W J 

(Cf. H 4.4.7], [31 p. 459]) 

Proposition 2 (Kemeny and Snell [4, 4.4.10]) Kemeny's constant is the 
trace of the resolvent Z: 

K = M lw = J2Z/. 

j 

Proof. 

M iw = ^2 MijW j 

= EV-E^ 
= EV. 

3 

using the fact that J2j =0. ■ 

This formula provides a computational verification that Kemeny's con- 
stant is constant, but doesn't explain why it is constant. Kemeny felt this 
keenly: A prize was offered for a more 'conceptual' proof, and awarded — 
rightly or wrongly — on the basis of the maximum principle argument outlined 
above. 

Still, there are advantages to having an explicit formula. For starters, 
the explicit formula reveals that the seek time is a spectral invariant of the 
matrix I — P: If we denote the eigenvalues of / — P by A = 0, Ai, . . . , \ n -i, 
then the eigenvalues of Z = (I - {P - P 00 ))- 1 - P°° are 0, . . . , and 

K = Tr(Z) = ^ + ... + -^-. 
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(Cf. Meyer [6], Aldous and Fill PQ, Levene and Loizou [5].) In terms of the 
eigenvalues a = 1, «i = 1 — A l5 . . . , a n _i = 1 — \ n -i of P we have 

K = TrZ = — - — + . . . + . 

1 — ol\ 1 — ct n -\ 

We'll have more to say about this later. 

Kemeny on the bus 

We now explore connections between the seek time, the famous bus paradox, 
and the Central Limit Theorem (CLT) for Markov Chains. 

Just to recall, the bus paradox is that for a Poisson process like radioactive 
decay, the expected length of the interval between the events that bracket 
any given instant is twice the average interval between events — so if buses 
are dispatched by a Geiger counter, you must expect to wait twice as long for 
the bus as you would if the buses came at regular intervals. The explanation 
for this is that any given instant is more likely to land in a long inter-event 
interval than in a short inter-event interval, so we're taking a weighted av- 
erage of the intervals between events, emphasizing the longer intervals, and 
this makes the expected waiting time longer than the average inter-event 
time. This inequality will be true for any renewal process in equilibrium: 
The factor of 2 disparity is special to the Poisson process, and arises because 
the Poisson process is memoryless and time-reversible. 

To make the connection of the seek time to the bus paradox, we think 
about the mean time from equilibrium to j: 

M wj ^ a-Mij. 

i 

Here we choose a starting state i at random according to w, and see how 
long it takes to get to j. This is backwards from what we did to define the 
seek time M iw , where we looked at the time to get from i to equilibrium. 

Unlike M iw , which is independent of i, the quantity M W j depends on the 
state j. Choosing the target state j at random according to w gets us back 
to K: 

j 
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Proposition 3 (Kemeny and Snell [4, 4.4.9]) The mean time from equi- 
librium to j is 

Proof. 

1 = */- 

w 3 J w 3 



M wj = Y, w l M l3 = 5>«(Z/ - Zi)- = Z/., 



i 



because J2i w% — 1 an d YL% w % Z i ] = 0. ■ 

To take advantage of this formula for M w j, and specifically, to use it to 
derive the CLT for Markov chains, we now recall a bit of renewal theory. (Cf. 
Feller [2 Chapter XIII].) 

A discrete renewal process is effectively just what you get if you watch a 
discrete-time Markov chain (possibly having infinitely many states) and take 
note of the times at which it is in some fixed state a. These times are called 
renewal times or renewals. The name derives from the fact that each time 
the chain reaches a it begins anew. The variables that tell the elapsed time 
between successive renewals are independent and identically distributed. 

So let X be a random variable whose distribution is that for the time 
between successive renewals, and let it have mean \i and variance a. We 
want to express the mean time r from equilibrium to the next renewal in 
terms of /i and a. 

Proposition (The bus equality). For a discrete renewal process with 
interarrival times having mean \x and variance a 2 , the mean time r from 
equilibrium to the next renewal is 

/i 2 + a 2 1 
2/i 2' 

The term — ~ here is an artifact of using discrete time. 
Proof. Let 

p{n) = Prob(X = n), 

so 

fj, = Exp(X) = 2J np(n) 

n 

and 

fi 2 + a 1 = Exp(X 2 ) = 5> 2 j9(n). 



6 



The expected time r from equilibrium to the next renewal is 

En np(n) 

1 fEn n2 p( n ) ,\ 



2 V En np(n) J 
{i 2 + a 2 1 
2/i 2" 

Corollary 4 (The bus inequality) 

u 1 

r > , 

~ 2 2' 

witt equality just if a = (a// interarrival times equal). ■ 

As in the bus equality above, the term — | here is an artifact of using 
discrete time. 

The bus equality shows that knowing the time r from equilibrium to the 
next renewal is equivalent to knowing a 2 , the variance of the renewal time: 



T 



fi 2 + a 2 1. 
2fi 2' 



a 2 = 2/ir + n — jj 2 . 

Of course the mean renewal time \x is involved here, too: We take that for 
granted. 

Now let's return to our Markov chain, and take for our renewal process 
visits to a given state j. The mean time between renewals is \x = — j. The 
expected time in equilibrium to the next renewal is r = M W j = Za 3 -^. But 
above we saw that 

a 2 = 2/ir + fi — /i 2 

so 

o> = 2ZJ -U + 4- ' 



3 (wi) 2 w 3 {w 3 ) 2 

Going back the other way, from \i and a 2 we can find (cf. Feller [21 (5.1) 
on p. 443]): 

■ a 2 - + /x 2 
j 2/i 2 
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Another piece of information about a renewal process that is equivalent 
to knowing a 2 or r (or Zj 3 , in the Markov chain case we just discussed) is 
the variance for the number of renewals over a long period, which shows up 
in the Central Limit Theorem for renewal processes: 

Theorem 5 (CLT for renewal processes) For a renewal process whose 
renewal time has mean \x and variance a 2 , the number of renewals over a 
long time T is approximately Gaussian with mean and variance Tj^. 

Idea of proof. To see a large number iV of renewals will take time 

T « Nfx ± \fNa. 

so the density of renewals over this interval is 




Thus 



Note. Feller [21 p. 341] gives the following formulas for Exp(iV) and 
Exp(iV 2 ), which he derives — or rather, asks readers to derive — using gener- 
ating functions: 

^ / t> t\ T + l a 2 - a - n 2 

Exp(iV) = + Jf—fL- + (1). 

fj, 2fi z 
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„ . (T + 2)(T+1) 2a 2 -2/i-/i2 
Exp(iV 2 ) = ± >-\ + + o{T). 



These combine to give 

2 

Var(iV) = ExpfiV 2 ) - Exp(N) 2 = °-T + o(T), 

which is the same as what we get from the CLT. 

The CLT for renewal processes translates into the following special case 
of the CLT for Markov chains (special because we are considering only the 
number of visits to one particular state, not the long-term average of a general 
function of the state). 

Corollary 6 For an ergodic Markov chain with resolvent Z , the number of 
visits to j over a long time is approximately Gaussian with mean T-^j and 
variance 

2 

T— = T{2Z 1 3 w ] + (w j f-w j ). ■ 

Grinstead and Snell (3 :) p. 466] attribute this formula for the variance to 
Frechet. 

Now just as in the case of the bus inequality, we get information from the 
fact that the variance here must be > 0: 

2Z/w j + (ir r f - w j > 0, 

so 

Summing over j gives an inequality for the seek time K: 
Proposition 7 

n — 1 



k = y,z/> 



J 



This inequality for the seek time was observed by Levene and Loizou 
[5]. They derived it from the fact that if the non-1 eigenvalues of P are 
1 < k < n — 1 then the non-1 eigenvalues of / — P + P°° are 1 — and the 
non-0 eigenvalues of Z = (I - P + P 00 )- 1 - P°° are X k = But the a fc 's 

lie in the unit disk, which maps to the region {x + iy : x > |} under the map 
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taking z to r^-, so the non-0 eigenvalues of Z have real part > |, and thus 
K, which is real, satisfies 

n-l ji-l i n—\ 



k=l k=l 



Taking stock. From the resolvent Z we've computed the variance of the 
return time, meaning the time get from a designated starting state j back 
to j. If instead we're interested in the variance of a hitting time, meaning 
the time to get from i to j ^ i, we'll need to look at Z 2 . We'd need this 
for the general CLT for Markov chains, which as noted above deals with the 
long-term average of a general function of the state of the chain, and requires 
knowing the covariance of the number of visits to a pair of states i and j. 
Looking beyond mean and variance, we get kth moments of return times from 
Z k ~ 1 and kth moments of hitting times from Z k . This was already in evidence 
for first moments: We need Z to find mean hitting times My = {Z? — Z?) 
whereas expected return times don't require knowing Z at all. 
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